Skip to content

feat(sketch_db): sid-level timeline_for_metric - #183

Merged
zzylol merged 1 commit into
mainfrom
feat/schema-retire-1-timeline
May 13, 2026
Merged

zzylol merged 1 commit into
mainfrom
feat/schema-retire-1-timeline

Conversation

@zzylol

@zzylol zzylol commented May 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Schema retirement #1 of 5. Adds sketch_db::query::timeline::timeline_for_metric(&SketchStore, metric, t1, t2) -> Vec<TimelineSegment> computed entirely from the sid catalog.

Schema/'s SchemaRegistry::timeline_for_metric stays alive in parallel; this PR just adds the sid-level version. Consumer migration (HTTP /api/v1/db/timeline, engine cross-reconfigure dispatch) is the next sub-PR.

Algorithm

  1. Snapshot SketchStore.instances for metric.
  2. Group by content signature (metric, agg_kind, group_by_keys). Many sids of one agg-config → one group.
  3. Per group, fold lifecycle fields: min(first_seen_unix_ms), retired-iff-all-retired, status precedence Active > Retired > Expired.
  4. Same segmenting + clipping as SchemaRegistry::timeline_for_metric.

TimelineSegment.agg_id carries a stable xxh64 of the content signature — same content-derived id idiom as compute_agg_config_id (PR #151).

Test plan

  • 7 new unit tests in sketch_db::query::timeline::tests.
  • cargo test -p data_plane --lib — 790/790 pass.

What's next

🤖 Generated with Claude Code

Schema retirement #1 of 5. Adds
`sketch_db::query::timeline::timeline_for_metric(&SketchStore, metric,
t1_ms, t2_ms) -> Vec<TimelineSegment>` that computes the per-metric
historical timeline entirely from the sid catalog
(`SketchStore::instances`).

Algorithm:

1. Snapshot all `SketchInstanceMetadata` for `metric` from the sid
   catalog.
2. Group by content signature `(metric, agg_kind, group_by_keys)`.
   Multiple sids sharing the same agg-config fold into one group.
3. Per group, fold the lifecycle fields: `min(first_seen_unix_ms)`,
   `Some(min(retired_at_ms))` iff every sid is retired,
   status = Active > Retired > Expired.
4. Apply the same segmenting + clipping as
   `SchemaRegistry::timeline_for_metric`.

`TimelineSegment.agg_id` now carries a stable xxh64 of the content
signature (the `(metric, agg_kind, group_by_keys)` tuple) — same
content-derived id idiom as `compute_agg_config_id` (PR #151). HTTP
callers see deterministic ids that don't depend on which specific
sid was first seen.

7 new unit tests cover: empty store, inverted range, single-active
signature, two-signatures-in-sequence, fold-of-many-sids-same-sig,
metric isolation, and signature-id determinism. Schema/'s
implementation stays alive in parallel until the consumers migrate.

Adds `SketchStore::snapshot_instances()` to support read-side scans.

783 + 7 = 790 lib tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit 8cd1476 into main May 13, 2026
zzylol added a commit that referenced this pull request May 13, 2026
Schema retirement #2 of 5. The `/api/v1/db/timeline` handler now
reads from `state.sketch_index` (always attached) via
`sketch_db::query::timeline::timeline_for_metric` (PR #183) instead
of `state.schemas.timeline_for_metric`.

User-visible behavior:

- Endpoint always works (no more 503 when schema-registry isn't
  wired — the sid catalog is always present).
- `agg_id` field carries a stable content-derived signature id
  (xxh64 of `metric + agg_kind + group_by_keys`) instead of the
  controller-emitted `agg_id` (which is gone after M2.2 / PR #152).
- Segments now reflect the sid catalog directly; reconfigure
  semantics propagate once the next sub-PR (lifecycle reconcile)
  lands.

Test impact:

- `test_get_timeline_without_registry_returns_503` rewritten to
  `..._with_no_sids_returns_empty_200` matching the new semantics.
- `test_get_timeline_returns_segments_after_reconfigure` ignored
  (uses POST /streaming-config → SchemaRegistry::reconcile, which
  doesn't yet propagate to the sid catalog; re-enabled in the next
  sub-PR with the lifecycle reconcile).

789/792 lib tests pass (3 ignored — 2 pre-existing + this one).

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request May 13, 2026
#185)

Schema retirement #3. Repoint `ASAPQueryEngine::timeline_for_query`
from `SchemaRegistry::timeline_for_metric` to the sid-level
`storage_engines::sketch_db::query::timeline::timeline_for_metric`
landed in #183. The cross-reconfigure dispatcher
(`try_handle_query_promql_via_timeline`) now reads its segments from
the sid catalog rather than from `SchemaRegistry`.

The sid-level timeline populates `TimelineSegment.agg_id` with a
content-hash of `(metric, agg_kind, group_by_keys)` rather than a
`StreamingConfig.aggregation_id`. Until schema retirement #5 ports
the per-segment dispatch to sid-level evaluation, the
segment-→-aggregation_config lookup inside the dispatcher is
best-effort: when no segment resolves to an in-config aggregation
the dispatcher returns `None` so the caller falls back to the
default single-agg path instead of regressing cross-reconfigure
queries to empty-result-plus-warnings.

The schema retirement plan keeps the `schema_registry` field on
`ASAPQueryEngine` alive for now — it's still referenced by the
ingest barrier and the swap-handler driver. Both go away in
retirements #4 + #5.

Two tests in `tests/schema_timeline_dispatch_tests.rs` are marked
`#[ignore]`: they build two distinct `AggregationConfig`s with
identical content (same metric / Sum / `host` grouping). In the
sid catalog those collapse to one signature group → one segment,
so the dispatcher can no longer reproduce the
schema-boundary-stitch scenario from a SchemaRegistry-shaped
fixture. The third single-schema regression test still passes
unchanged. Re-enabling these is part of retirement #5 (sid-level
dispatch) or a fixture rewrite that uses two genuinely distinct
signatures.

787/787 lib tests pass; 5 ignored.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol deleted the feat/schema-retire-1-timeline branch July 17, 2026 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant